Fundamentals of Missing Data in Evaluation

Presentation to MSU Department of Psychology, Program Evaluation Occasional Speaker Series, East Lansing, MI

Steven J. Pierce

pierces1@msu.edu

Center for Statistical Training and Consulting

2024-12-05

Outline

What is missing data?
Why do we end up with missing data?
Why should we care about missing data?
How can we diagnose the missing data issues for a given study?
What should we do about missing data?

What is missing data?

Missing data (MD) are measurements you want or intended to collect but did not get.^[1]

Having MD is common in research & evaluation studies.
If you do much evaluation work, you will run into MD.

Why do we end up with missing data?

Data collection doesn’t always go according to plan…

Human Factors	Other Factors
Participant behavior	Equipment failures
Evaluator errors	Records/Databases
Partner behavior	Unusual Events

Like many things in life, data collection doesn’t always go according to plan. > Human factors often contribute to the occurrence of missing data. For example, a survey participant may skip a question without recording any response, offer an illegible or irrelevant answer, refuse to answer it, or simply say “I don’t know”. They may skip whole instruments or entire data collection visits. But evaluators occasionally make errors that lead to missing data. Imagine forgetting to take a data collection form with you, or programming the skip pattern for an online survey incorrectly. Sometimes, our evaluation partners’ decisions or behavior can prevent obtaining some data we wanted. Perhaps they object to sharing something with you, or were not diligent about logging some events of interest.

Other things can go wrong too. Perhaps a physical instrument breaks, or its battery dies, or your online survey app crashes. If you’re using an organization’s records or databases, those may be incomplete, difficult to search for relevant data, coded in unhelpful ways, or could have been lost or damaged or even destroyed according to some retention policy. In terms of events, consider the recent hurricanes in Florida that disrupted data collection for one of my projects.

Missing Data & Project Lifecycle

Why should we care about missing data?

Ethics for Evaluators

Handling missing data well enacts our guiding principles^[2]:

AEA logo.

Systematic inquiry
Competence
Integrity

Perhaps the broadest answer to why we should care about missing data is that this is an ethics issue. Thoughtful handling of missing data is a good way to enact several of AEA’s guiding principles for evaluators.

> Systematic inquiry demands that we examine the quality of the available data and use it appropriately. We have an obligation to use sound methodology. There are scientific consequences of missing data but there are also legitimate, evidence-based methods for minimizing its impact on our analyses, conclusions, and products.
> Competence demands that we invest in skills necessary to understand and handle missing data properly because doing it poorly may compromise our evaluations. Our funders, clients, and stakeholders–including the program staff and the people and communities they serve and operate within–need and deserve competent evaluation services.
> Integrity demands that we be transparent about the amount of missing data, how we handled it, and how it may affect our evaluation results.

Scientific Activities^[3]

There are 3 major scientific activities that can be affected by missing data.

Making structured observations of constructs.
Using observations to draw inferences about relationships between constructs.
Generalizing the results to populations beyond the collected sample.

Consequences for Measurement^[3]

Availability of constructs
Decreased reliability due to increased error variance
Bias from poor content coverage
Construct validity

Missing data can affect the availability of a construct for your analyses. The extreme case would be a construct measured by a single variable. If that’s missing, then the entire construct is missing for that individual and it may be very difficult to recover that information. Missing data is still a problem when we use multiple indicators to measure a construct, though perhaps less severe, and it may depend on how you combine items to create the score.

Missing data decreases the reliability and construct validity of our measures. Internal consistency reliability decreases because missing data increases the error variance in a measure. You may recall that reliability is a function of how many items are used, so fewer observed items leads to lower reliability.

Further, some items may be more important indicators of a construct than others. You may get a biased estimate of the construct due to poorer content coverage if they are missing. Construct validity depends on both reliability and content coverage, so missing data is a threat to validity.

Consequences for Internal Validity^[1,3]

Selection bias
Compromised randomization
Power and precision
Inaccurate model assumptions

Missing data is a threat to a study’s internal validity for several reasons. Selection bias can result from systematic differences between subgroups of participants on constructs relevant to the study conclusions. For example, those subgroups could be treatment versus control groups, people who complete the study versus drop out, or those who have complete data versus incomplete data. These differences may manifest with respect to demographic variables, covariates, focal predictors, mediators, moderators, or outcomes. Imagine evaluating a longitudinal health study where people who start out with worst symptoms drop out before the posttest more often than those with mild symptoms. Dropping out leads to missing data at posttest. If we fail to handle that properly in the analysis, we may draw inaccurate conclusions about the longitudinal trend in outcome scores. Or perhaps the retention rates differ between treatment and control groups in that same study. Selection bias from such differences reduce the sample representativeness and become rival explanations for findings, making them a threat to internal validity.

We use randomization to make groups equivalent on measured and unmeasured confounding variables as a way to strengthen validity. That works best in large samples. However, missing data can compromise that effect of randomization by reducing sample size. It can also reduce randomization’s ability to achieve equivalence between groups on confounding variables when data are missing differentially across the groups. That reduces internal validity.

Most statistical software defaults to listwise deletion of cases that have any missing values on the variables involved in an analysis. Reducing the sample size will always reduce both power to detect effects and the precision of effect size estimates. In addition to reducing overall sample size, missing data may lead to more unbalanced sample sizes between groups in a study, which also reduces statistical power in many models.

Finally, missing data can change the distributions of variables in ways that invalidate the assumptions of various statistical models. Some models are more robust than others, but making inaccurate modeling assumptions risks running analyses that will yield misleading conclusions. That’s another threat to internal validity.

Consequences for Generalizability^[1,3]

A representative sample is crucial to generalizing to the intended population!

Theory development & cumulative knowledge
Policy & decision-making

We draw samples to study populations when it isn’t feasible to collect the data we want from the entire population. The key to getting inferences that properly generalize from a sample to the intended population is getting a representative sample. Biases creep into findings when we use non-representative samples. Missing data can introduce systematic biases into our samples. Suppose you’re studying a behavior that’s illegal or stigmatized in some way. Some people who engage in it may be reluctant to provide all the data you want. If we ignore the missing data problem, listwise deletion will result in analyzing the subset of the intended population that’s willing to answer our questions. That in effect changes the population to which we could legitimately generalize our findings from “people who engage in behavior x” to “people who engage in behavior x and provided complete data”. Many people fail to attend to that though and assume their results still apply to the original intended population.

That has consequences for developing and testing scientific theories and accumulating knowledge across studies. An example is the effect of publication bias, also called the file drawer problem, on meta-analyses. We know that studies with non-significant findings are less likely to get published. That means the published literature has a missing data problem that can seriously distort the results of meta-analyses that aim to summarize results of a collection of published papers.

Finally, the big picture here is that when missing data impair generalizability, we may be using inaccurate or misleading findings to inform public policy and important decisions.

How can we diagnose the missing data issues for a given study?

Cattell’s Data Box^[4]

How much data is there? Data volume is \(N_{values} = P \times V \times T\)

Slices through the cube represent subsets of data.
Constructs are often measured by groups of adjacent variables (items).
Missing data puts holes in your cube!

Raymond Cattell described data as forming a 3-dimensional box or cube, with the dimensions being persons, variables, and occasions (or times). This is useful for thinking about the total volume of data we aim to collect. At least for a simple longitudinal study where we collect the same instruments for each person at each time point, we can just multiply the number of desired participants by the number of variables by the number of time points to get the number of desired data values, which is the volume of the cube. Different slices through the cube represent specific subsets of data. The top layer would be all the data for a given person (all variables at all time points). The left-most vertical slice would be all the data for a specific variable across all persons and time points. The front face of the cube would be time 1 data on all variables for all persons. We could further refine this idea by recognizing that constructs are often measured by groups of adjacent variables (items).

Missing data puts holes in the cube, turning it into something like Swiss cheese. The impact of that depends on how much data volume is lost and any patterns in where and why those holes are located relative to the data you want to analyze.

Types of Missingness^[3,5]

Item level
Construct level
Person level (unit non-response)
Person-period level (wave nonresponse; intermittent vs. dropout)

There are a variety of types of missingness to consider as we begin describing a given data set. One place to start involves examining different parts of the data cube. For example, item-level missingness pertains to an individual variable. What percentage of its values are missing? Contrast that with construct-level missingness, which is when an entire construct is missing. Some scoring methods leave a construct missing if any of its items are missing, others may still yield a score despite some missing items. Construct missingness is usually a bigger problem than item-level missing data.

Person-level missing data, some times called unit nonresponse, is when all the data for a person is missing. Refusing to participate will usually cause person-level missing data. That’s an extreme example, but a related type of missingness happens at the person-period level when a participant skips an entire data collection event. That’s also called wave non-response. It affects all constructs collected at that time point. In longitudinal studies, person-period missingness could be intermittent, with a participant skipping one event but returning for subsequent events, or it may be a result of dropout or attrition that affects all remaining time points.

Describing the Amount of MD^[3,6]

Report numbers & percentages of:

Participants w/ any data at each time point (retention/attrition)
Complete vs. incomplete cases (overall & by time point)
Missing values for each variable & construct
Reasons for attrition/dropout and other missing data

The dimensions of Cattell’s data box suggests some obvious summaries of the amount of missing data. For example, we probably want to examine and report the number and percentage of participants in the study who have any data at each time point and use that to examine retention or attrition rates. Then we can go further and report the number of complete vs. incomplete cases, both overall and separately by time point. Then, we can compute the numbers and percentages of missing values for each variable, both overall and separately by time point.

Finally, it can be really useful to report on the reasons why participants drop out, skip data collection events, or have other missing data.

All of these things get easier when you have a tracking system that records:

Whether & when participants attended data each collection event
When & why participants exit the study

Patterns of Missing Data^[5]

Y: matrix of all the values that could be observed
Y_obs: subset of Y values that end up observed
Y_miss: subset of Y values that end up missing
R: response matrix of dummy-coded missingness indicators showing which Y values are observed (0, part of Y_obs) vs. missing (1, part of Y_miss)

Tip

We can aggregrate and visualize R to describe patterns of missingness!

Example Patterns of Missing Data

Missingness patterns for Dutch boys growth study data (748 boys, 9 variables, 1 time point)^[7]

This is graph summarizes data from a developmental study about growth in 748 Dutch boys. The variables include age, region, weight, height, body mass index, head circumference, and three measures of pubertal development. This is just a convenient dataset because it comes with an R package that I use to deal with missing data. The graph illustrates what I mean by missing data patterns. Each row in the graph represents a distinct pattern of observed versus missing values across the whole set of variables. The left side is annotated with the number of boys showing each pattern, the right side tells you how many missing values there are on that row, and the bottom shows the total number of missing values for each variable. With 748 boys and 9 variables, there should be 6732 values, but we can see at the bottom that 1622 (24%) are missing. So this data box has a lot of holes in it.

The first row shows 223 boys have complete data: that’s about 30% of the sample. So, 70% of them have at least one missing value. We can also see that the most common pattern is that 437 boys have missing data for the 3 pubertal development measures, but observed data on all the other measures. That’s 58% of the sample.

Another thing we can see here is that body mass index is always missing whenever either height or weight is missing. That’s because BMI is a measure derived from both height and weight. That’s construct missingness caused by item-missingness.

Rubin’s Mechanisms of Missingness^[8]

Missing completely at random (MCAR)
Missing at random (MAR)
Missing not at random (MNAR)

Impact on Statistical Results^[3]

Some mechanisms yield more bias: MCAR < MAR < MNAR

Understanding Donald Rubin’s system for classifying missing data is crucial to making good decisions about how to handle missing data. This classification systm is pervasive in the literature on missing data because it is very useful despite having unintuitive names. Rubin defined three categories, each referring to the probability of missing values given information about the variables with missing data, other covariates, and a hypothetical mechanism that causes the missing data. The three categories are:

Missing completely at random (MCAR)
Missing at random (MAR)
Missing not at random (MNAR)

These three mechanisms affect the amount of bias missing data causes in statistical analyses, with MCAR yielding the least bias and MNAR yielding the most. Let’s take a closer look at each mechanism.

MCAR

MCAR is when neither observed nor unobserved values predict which values are missing.

MAR

MAR is when observed values predict which values are missing.

MNAR

MNAR is when unobserved values predict which values are missing.

Predictors of Attrition & Missingness in Longitudinal Studies

Study arm: Compare attrition rates
Study site (in multisite studies)
Baseline/pretest values of outcome variables may predict who drops out or has missing values
Other covariates (demographics)

What should we do about missing data?

Prevention

An ounce of prevention is better than a pound of cure

^[9],^[10]

Treatment

Reporting

Advice

Collaborate with a statistician!

Practical Options

Item-level missingness in scale scores^[11,12]

References

1. Fernández-García, M. P., Vallejo-Seco, G., Livácic-Rojas, P., & Tuero-Herrero, E. (2018). The (ir)responsibility of (under)estimating missing data. Frontiers in Psychology, 9(556). https://doi.org/10.3389/fpsyg.2018.00556

2. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles

3. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.

4. Cattell, R. B. (1966). The data box: Its ordering of total resources in terms of possible relations systems. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 67–128). Rand McNally.

5. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Bulletin, 7(2), 147–777. https://doi.org/10.1037//1082-989X.7.2.I47

6. van Buuren, S. (2018). Flexible imputation of missing data (2nd ed.). Chapman & Hall/CRC Press. https://doi.org/10.1201/9780429492259

7. Fredriks, A. M., Buuren, S. van, Burgmeijer, R. J. F., Meulmeester, J. F., Beuker, R. J., Brugman, E., Roede, M. J., Verloove-Vanhorick, S. P., & Wit, J.-M. (2000). Continuing positive secular growth change in the netherlands 1955-1997. Pediatric Research, 47, 316–323. https://doi.org/10.1203/00006450-200003000-00006

8. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581

9. Leeuw, E. D. de. (2001). Reducing missing data in surveys: An overview of methods. Quality & Quantity, 35(2), 147–160. https://doi.org/10.1023/A:1010395805406

10. Wisniewski, S. R., Leon, A. C., Otto, M. W., & Trivedi, M. H. (2006). Prevention of missing data in clinical research studies. Biological Psychiatry, 59, 997–1000. https://doi.org/10.1016/j.biopsych.2006.01.017

11. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530

12. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590